Comparing an Integrated QA system performance - A Preliminary Model

A complex NLP application such as a question answering system requires sophisticated methods for evaluation, especially if there is a mechanism to submit factoid type queries as well as natural language (NL)-type questions. As part of our research we created an integrated tool that allows users to search two ways – using a keyword query and using a NL question. We present a method to evaluate output from the two types of searches and we explore the correlation of scores when judgments of output are made against keywords and those made against NL questions. We present a preliminary model, which may serve as a guideline to evaluate similar systems’ performance.